Dynamical multilayer neural networks that learn continuous trajectories

The feed-forward multilayer networks (perceptrons, radial basis function networks (RBF), probabilistic networks, etc.) are currently used as „static systems“ in pattern recognition, speech generation, identification and control, prediction, etc. (see, e. g. [1]). Theoretical works by several researchers, including [2] and [3] have proved that, even with one hidden layer, a perceptron neural network can uniformly approximate any continuous function over a compact domain, provided that this perceptron has a sufficient number of neurons in the hidden layer. Recently, interest has increased to apply the neural networks in learning continuous trajectories, identification and/or control of dynamical systems. In such a case it is natural to use the networks involving dynamical elements in the form of feedback connections which are known as recurrent neural networks. Although a huge amount of work and studies in the field of feed-forward neural networks has been done, only few of the results concern learning continuous trajectories by means of dynamical recurrent neural networks. Pearlmutter [4], [5], Werbos [6], Toomarian and Barhen [7] have developed an algorithm using gradient-descent-based methods which is known as back-propagation through time (BPTT); it learns time-varying external inputs and produces either desired temporal behaviours over a bounded time interval or trains non-fixed-point attractors. Williams and Zipser [8], Meert and Ludik [9] have constructed a gradient descent learning rule which they call real-time recurrent learning (RTRL) and which can deal with time sequences of arbitrary length. A stochastic search method based on an adaptive simulated annealing algorithm has been used by Cohen et al. [10] to efficiently train recurrent neural networks with time delays (TDRNN). An effort was made in the above investigation to implement several benchmark tasks using minimum size networks. In all the above investigations, a dynamical neural network evolves in accordance with the following general equations